Selecting During A System Shutdown Procedure, A Restart Incident Checkpoint Of An Incident Analyzer In A Distributed Processing System

ABSTRACT

Methods, apparatuses, and computer program products for selecting during a system shutdown procedure, a restart incident checkpoint of an incident analyzer in a distributed processing system. Embodiments include the incident analyzer determining whether at least one incident is in a queue. If at least one incident is in the queue, the incident analyzer selects as the restart incident checkpoint, a last incident completed checkpoint. If at least one incident is not in the queue, the incident analyzer determines whether the last incident completed checkpoint matches a last incident analysis pool selection checkpoint. If the last incident completed checkpoint matches a last incident analysis pool selection checkpoint, the incident analyzer selects as the restart incident checkpoint, a monitor checkpoint. If the last incident completed checkpoint does not match the last incident analysis pool selection checkpoint, the incident analyzer selects as the restart incident checkpoint, the last incident completed checkpoint.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under Contract No.HR0011-07-9-0002 awarded by the Defense Advanced Research ProjectsAgency (DARPA). The Government has certain rights in this invention.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention is data processing, or, more specifically,methods, apparatuses, and computer program products for selecting duringa system shutdown procedure, a restart incident checkpoint of anincident analyzer in a distributed processing system.

2. Description of Related Art

The development of the EDVAC computer system of 1948 is often cited asthe beginning of the computer era. Since that time, computer systemshave evolved into extremely complicated devices. Today's computers aremuch more sophisticated than early systems such as the EDVAC. Computersystems typically include a combination of hardware and softwarecomponents, application programs, operating systems, processors, buses,memory, input/output devices, and so on. As advances in semiconductorprocessing and computer architecture push the performance of thecomputer higher and higher, more sophisticated computer software hasevolved to take advantage of the higher performance of the hardware,resulting in computer systems today that are much more powerful thanjust a few years ago.

Modern distributed processing systems for intensive computing may havemillions of devices with many processes running on each device all ofwhich are capable of error and status reporting for automated errorrecovery, reporting to a systems administrator, and for other reasons.In many cases, in the case of an error for example, the sheer number ofsuch error reports and status reports are so overwhelming that theycannot be handled in a meaningful manner. For example, a systemsadministrator receiving a hundred thousand error reports may beoverwhelmed by the sheer number of such reports and therefore in theaggregate those reports become more and more unhelpful and irrelevant.

SUMMARY OF THE INVENTION

Methods, apparatuses, and computer program products for selecting duringa system shutdown procedure, a restart incident checkpoint of anincident analyzer in a distributed processing system. Embodimentsinclude the incident analyzer determining whether at least one incidentis in an incident receive queue of the incident analyzer. If at leastone incident is in the queue, the incident analyzer selects as therestart incident checkpoint, a last incident completed checkpoint. If atleast one incident is not in the queue, the incident analyzer determineswhether the last incident completed checkpoint matches a last incidentanalysis pool selection checkpoint. If the last incident completedcheckpoint matches a last incident analysis pool selection checkpoint,the incident analyzer selects as the restart incident checkpoint, amonitor checkpoint. If the last incident completed checkpoint does notmatch the last incident analysis pool selection checkpoint, the incidentanalyzer selects as the restart incident checkpoint, the last incidentcompleted checkpoint.

The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescriptions of exemplary embodiments of the invention as illustrated inthe accompanying drawings wherein like reference numbers generallyrepresent like parts of exemplary embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary system for selecting during a systemshutdown procedure, a restart incident checkpoint of an incidentanalyzer in a distributed processing system according to embodiments ofthe present invention.

FIG. 2 sets forth a block diagram of automated computing machinerycomprising an exemplary computer useful in selecting during a systemshutdown procedure, a restart incident checkpoint of an incidentanalyzer in a distributed processing system according to embodiments ofthe present invention.

FIG. 3 sets forth a block diagram of an exemplary system for selectingduring a system shutdown procedure, a restart incident checkpoint of anincident analyzer in a distributed processing system in a distributedprocessing system according to embodiments of the present invention.

FIG. 4 sets forth a diagram illustrating assigning events to an eventpool according to embodiments of the present invention.

FIG. 5 sets forth a diagram illustrating assigning alerts to an alertpool according to embodiments of the present invention.

FIG. 6 sets forth a flow chart illustrating an example method ofselecting during a system shutdown procedure, a restart incidentcheckpoint of an incident analyzer in a distributed processing systemaccording to embodiments of the present invention.

FIG. 7 sets forth a flow chart illustrating an additional method ofselecting during a system shutdown procedure, a restart incidentcheckpoint of an incident analyzer in a distributed processing systemaccording to embodiments of the present invention.

FIG. 8 sets forth a flow chart illustrating an additional method ofselecting during a system shutdown procedure, a restart incidentcheckpoint of an incident analyzer in a distributed processing systemaccording to embodiments of the present invention.

FIG. 9 sets forth a flow chart illustrating an additional method ofselecting during a system shutdown procedure, a restart incidentcheckpoint of an incident analyzer in a distributed processing systemaccording to embodiments of the present invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Exemplary methods, apparatuses, and computer program products forselecting during a system shutdown procedure, a restart incidentcheckpoint of an incident analyzer in a distributed processing systemaccording to embodiments of the present invention are described withreference to the accompanying drawings, beginning with FIG. 1. FIG. 1illustrates an exemplary system for selecting during a system shutdownprocedure, a restart incident checkpoint of an incident analyzer in adistributed processing system according to embodiments of the presentinvention. A distributed processing system is typically implemented asmultiple autonomous or semi-autonomous computers that communicatethrough a computer network. In such example distributed processingsystems, the computers often interact with each other in order toachieve a common goal. A computer program that runs in such an exampledistributed system is typically called a distributed program, anddistributed programming is often used to describe the process of writingsuch programs.

In the example of FIG. 1, the distributed processing system (101) isimplemented as a parallel computer (100), non-volatile memory for thecomputer in the form of data storage device (118), an output device forthe computer in the form of a printer (120), and an input/output devicefor the computer in the form of a computer terminal (122). The parallelcomputer (100) in the example of FIG. 1 also includes a plurality ofcompute nodes (102). Each compute node is an automated computing devicecomposed of one or more computer processors, its own computer memory,and its own input/output functionality. The compute nodes (102) arecoupled for data communications by several independent datacommunications networks including a high speed Ethernet network (174), aJoint Test Action Group (‘JTAG’) network (104), a tree network (106)which is optimized for collective operations, and a torus network (108)which is optimized for point to point operations. The tree network (106)is a data communications network that includes data communications linksconnected to the compute nodes so as to organize the compute nodes as atree. Each data communications network is implemented with datacommunications links among the compute nodes (102). The datacommunications links provide data communications for parallel operationsamong the compute nodes of the parallel computer.

In addition to compute nodes, computer (100) includes input/output(‘I/O’) nodes (110, 114) coupled to compute nodes (102) through one ofthe data communications networks (174). The I/O nodes (110, 114) provideI/O services between compute nodes (102) and I/O devices (118, 120,122). I/O nodes (110, 114) are connected for data communications throughlocal area network (‘LAN’) (130). Computer (100) also includes a servicenode (116) coupled to the compute nodes through one of the networks(104). The service node (116) provides service common to pluralities ofcompute nodes, loading programs into the compute nodes, starting programexecution on the compute nodes, retrieving results of program operationson the computer nodes, and so on. The service node (116) runs an eventand alert analysis module (124) and communicates with users (128)through a service application interface (126) that runs on the computerterminal (122).

Many of the components of the distributed processing system of FIG. 1,that is the devices of the distributed processing system or processesrunning on the devices of the distributed processing system of FIG. 1are capable of some form of error or status reporting through events andmany of such components are also capable of receiving alerts in responseto one or more of such events. Often in distributed processing systemshundreds of thousands or millions of components may provide or receiveincidents, often in the form of events or alerts.

An incident is a generic term used in this specification to mean anidentification or notification of a particular occurrence on a componentof a distributed processing system such as events described below, arefined identification of an occurrence often based on events such as analert described below, or other notifications as will occur to those ofskill in the art.

Incidents are administered in pools for event and alert analysisaccording to embodiments of the present invention. A pool of incidentsis a collection of incidents organized by the time of either theiroccurrence, by the time they are logged in an incident queue, includedin the pool, or other time as will occur to those of skill in the art.That is, incident pools are a collection of incidents organized by time.Such incident pools often provide the ability to analyze a group of timerelated incidents. Often such incident pools are useful in identifyingfewer and more relevant incidents in dependence upon multiple relatedincidents.

An event according to embodiments of the present invention is anotification of a particular occurrence in or on a component of thedistributed processing system. Such events are sent from the componentupon which the occurrence occurred or another reporting component to anevent and alert analysis module according to the present invention.Often events are notifications of errors occurring in a component of thedata processing system. Events are often implemented as messages eithersent through a data communications network or shared memory. Typicalevents for event and alert analysis according to embodiments of thepresent invention include attributes such as an occurred time, a loggedtime, an event type, an event ID, a reporting component, and a sourcecomponent, and other attributes.

An alert according to embodiments of the present invention is refinedidentification of an occurrence, such and an error based upon more thanone event and therefore provides an identification of the occurrence inthe context of its operation in the distributed processing system. Oftenan alert may be a notification of a particular error type of occurrencethat is identified in dependence upon the plurality of events receivedfrom one or more components of the data processing system, such as, forexample, a link failure among a plurality of devices each of which areproducing many events based upon the single link failure, or a powerfailure provoking thousands of events, and so on.

Alerts are often implemented as messages to be sent through a datacommunications network or shared memory. Typical alerts according toembodiments of the present invention have attributes attached to thembased upon the attributes of the events received from which they areidentified.

The service node (116) of FIG. 1 has installed upon it an event andalert analysis module (124) that includes at least two incidentanalyzers implemented as event analyzers and alert analyzers capable ofselecting during a system shutdown procedure, a restart incidentcheckpoint of an incident analyzer in a distributed processing systemaccording to embodiments of the present invention.

Specifically, the event and alert analysis module (124) is implementedas automated computing machinery configured so that during a systemshutdown procedure, the incident analyzer determines whether at leastone incident is in an incident receive queue of the incident analyzer.If at least one incident is in the queue, the incident analyzer selectsas the restart incident checkpoint, a last incident completedcheckpoint. If at least one incident is not in the queue, the incidentanalyzer determines whether the last incident completed checkpointmatches a last incident analysis pool selection checkpoint. If the lastincident completed checkpoint matches a last incident analysis poolselection checkpoint, the incident analyzer selects as the restartincident checkpoint, a monitor checkpoint. If the last incidentcompleted checkpoint does not match the last incident analysis poolselection checkpoint, the incident analyzer selects as the restartincident checkpoint, the last incident completed checkpoint.

The arrangement of nodes, networks, and I/O devices making up theexemplary distributed processing system illustrated in FIG. 1 are forexplanation only, not for limitation of the present invention.Distributed data processing systems configured to select during a systemshutdown procedure, a restart incident checkpoint according toembodiments of the present invention may include additional nodes,networks, devices, and architectures, not shown in FIG. 1, as will occurto those of skill in the art. The parallel computer (100) in the exampleof FIG. 1 includes sixteen compute nodes (102); parallel computersconfigured to select during a system shutdown procedure, a restartincident checkpoint according to embodiments of the present inventionsometimes include thousands of compute nodes. In addition to Ethernetand JTAG, networks in such data processing systems may support many datacommunications protocols including for example TCP (Transmission ControlProtocol), IP (Internet Protocol), and others as will occur to those ofskill in the art. Various embodiments of the present invention may beimplemented on a variety of hardware platforms in addition to thoseillustrated in FIG. 1.

Selecting during a system shutdown procedure, a restart incidentcheckpoint of an incident analyzer in a distributed processing system inaccordance with the present invention is generally implemented withcomputers, that is, with automated computing machinery. In the system ofFIG. 1, for example, all the service nodes, I/O nodes, compute nodes, ofthe parallel computer are implemented to some extent at least ascomputers. For further explanation, therefore, FIG. 2 sets forth a blockdiagram of automated computing machinery comprising an exemplarycomputer (152) useful in selecting during a system shutdown procedure, arestart incident checkpoint according to embodiments of the presentinvention. The computer (152) of FIG. 2 includes at least one computerprocessor (156) or ‘CPU’ as well as random access memory (168) (‘RAM’)which is connected through a high speed memory bus (166) and bus adapter(158) to processor (156) and to other components of the computer (152)and through an expansion bus to adapters for communications with othercomponents of a distributed processing system (101).

Stored in RAM (168) is an event and alert analysis module (124), amodule of automated computing machinery for selecting during a systemshutdown procedure, a restart incident checkpoint according toembodiments of the present invention. The event and alert analysismodule (124) includes two incident analyzers, a monitor (204), and acheckpoint manager (299) according to embodiments of the presentinvention.

The checkpoint manager (299) and the monitor (204) select during asystem shutdown procedure, a restart incident checkpoint according toembodiments of the present invention by detecting the restart incidentcheckpoint and sending to the plurality of incident analyzers, a streamof incidents beginning with the incident identified by the restartincident checkpoint and continuing with any incidents having a neweridentification number than the incident identified by the restartincident checkpoint.

The incident analyzers include an event analyzer (208) and an alertanalyzer (218). The event analyzer of FIG. 2 is a module of automatedcomputing machinery capable of identifying alerts in dependence uponreceived events. That is, event analyzers typically receive events andproduce alerts. In many embodiments, a plurality of event analyzers areimplemented in parallel. Often such event analyzers are assigned to aparticular pool of events and may be focused on events from a particularcomponent or caused by a particular occurrence to produce a more conciseset of alerts.

The alert analyzer (218) of FIG. 2 is a module of automated computingmachinery capable of identifying alerts for transmission from events andother alerts, identifying additional alerts for transmission, andsuppressing unnecessary, irrelevant, or otherwise unwanted alertsidentified by the event analyzer. That is, alert analyzers typicallyreceive alerts and events and produce or forward alerts in dependenceupon those alerts and events. In many embodiments, a plurality of alertanalyzers are implemented in parallel. Often such alert analyzers areassigned to a particular pool of alerts and may be focused on alertswith particular attributes to produce a more concise set of alerts.

In addition to the general functions described above, the event analyzer(208) and the alert analyzer (218) may each be configured to selectduring a system shutdown procedure, a restart incident checkpoint in adistributed processing system according to embodiments of the presentinvention. An event analyzer selects during a system shutdown procedure,a restart incident checkpoint according to embodiments of the presentinvention by determining whether at least one event is in an eventreceive queue of the event analyzer. If at least one event is in thequeue, the event analyzer selects as the restart event checkpoint, alast event completed checkpoint. If at least one event is not in thequeue, the event analyzer determines whether the last event completedcheckpoint matches a last event analysis pool selection checkpoint. Ifthe last event completed checkpoint matches a last event analysis poolselection checkpoint, the event analyzer selects as the restart eventcheckpoint, a monitor checkpoint. If the last event completed checkpointdoes not match the last event analysis pool selection checkpoint, theevent analyzer selects as the restart event checkpoint, the last eventcompleted checkpoint.

An alert analyzer selects during a system shutdown procedure, a restartincident checkpoint according to embodiments of the present invention bydetermining whether at least one alert is in an alert receive queue ofthe alert analyzer. If at least one alert is in the queue, the alertanalyzer selects as the restart alert checkpoint, a last alert completedcheckpoint. If at least one alert is not in the queue, the alertanalyzer determines whether the last alert completed checkpoint matchesa last alert analysis pool selection checkpoint. If the last alertcompleted checkpoint matches a last alert analysis pool selectioncheckpoint, the alert analyzer selects as the restart alert checkpoint,a monitor checkpoint. If the last alert completed checkpoint does notmatch the last alert analysis pool selection checkpoint, the alertanalyzer selects as the restart alert checkpoint, the last alertcompleted checkpoint.

Also stored in RAM (168) is an operating system (154). Operating systemsuseful for relevant alert delivery according to embodiments of thepresent invention include UNIX™, Linux™, Microsoft XP™, AIX™, IBM'si5/OS™, and others as will occur to those of skill in the art. Theoperating system (154), event and alert analysis module (124), the eventanalyzer (208), the alert analyzer (218) in the example of FIG. 2 areshown in RAM (168), but many components of such software typically arestored in non-volatile memory also, such as, for example, on a diskdrive (170).

The computer (152) of FIG. 2 includes disk drive adapter (172) coupledthrough expansion bus (160) and bus adapter (158) to processor (156) andother components of the computer (152). Disk drive adapter (172)connects non-volatile data storage to the computer (152) in the form ofdisk drive (170). Disk drive adapters useful in computers for selectingduring a system shutdown procedure, a restart incident checkpointaccording to embodiments of the present invention include IntegratedDrive Electronics (‘IDE’) adapters, Small Computer System Interface(‘SCSI’) adapters, and others as will occur to those of skill in theart. Non-volatile computer memory also may be implemented for as anoptical disk drive, electrically erasable programmable read-only memory(so-called ‘EEPROM’ or ‘Flash’ memory), RAM drives, and so on, as willoccur to those of skill in the art.

The example computer (152) of FIG. 2 includes one or more input/output(‘I/O’) adapters (178). I/O adapters implement user-orientedinput/output through, for example, software drivers and computerhardware for controlling output to display devices such as computerdisplay screens, as well as user input from user input devices (181)such as keyboards and mice. The example computer (152) of FIG. 2includes a video adapter (209), which is an example of an I/O adapterspecially designed for graphic output to a display device (180) such asa display screen or computer monitor. The video adapter (209) isconnected to processor (156) through a high speed video bus (164), busadapter (158), and the front side bus (162), which is also a high speedbus.

The exemplary computer (152) of FIG. 2 includes a communications adapter(167) for data communications with other computers (182) and for datacommunications with a data communications network (100). Such datacommunications may be carried out serially through RS-232 connections,through external buses such as a Universal Serial Bus (‘USB’), throughdata communications data communications networks such as IP datacommunications networks, and in other ways as will occur to those ofskill in the art. Communications adapters implement the hardware levelof data communications through which one computer sends datacommunications to another computer, directly or through a datacommunications network. Examples of communications adapters useful forselecting during a system shutdown procedure, a restart incidentcheckpoint according to embodiments of the present invention includemodems for wired dial-up communications, Ethernet (IEEE 802.3) adaptersfor wired data communications network communications, and 802.11adapters for wireless data communications network communications.

For further explanation, FIG. 3 sets forth a block diagram of anexemplary system for selecting during a system shutdown procedure, arestart incident checkpoint and relevant alert delivery in a distributedprocessing system (102) according to embodiments of the presentinvention. The system of FIG. 3 includes an event and alert analysismodule (124). The event and alert analysis module (124) of FIG. 3receives in an event queue (206) a plurality of events (202) from one ormore components of a distributed processing system (102). A component ofa distributed processing system according to embodiments of the presentinvention may be a device of the distributed processing system or aprocess running on a device of the distributed processing. Suchcomponents are often capable of some form of event transmission, oftenfor error or status reporting.

An event according to embodiments of the present invention is anotification of a particular occurrence in or on a component of thedistributed processing system. Such events are sent from the componentupon which the occurrence occurred or another reporting component to anevent and alert analysis module according to the present invention.Often events are notifications of errors occurring in a component of thedata processing system. Events are often implemented as messages eithersent through a data communications network or shared memory. Typicalevents for event and alert analysis according to embodiments of thepresent invention include attributes such as an occurred time, a loggedtime, an event type, an event ID, a reporting component, and a sourcecomponent, and other attributes. An occurred time is the time at whichthe event occurred on the component. A logged time is the time the eventwas included in the event queue (206) and is typically inserted into theevent by a monitor. An event type is a generic type of event such as forexample, power error, link failure error, errors related to notreceiving messages or dropping packets and so on as will occur to thoseof skill in the art. An event ID is a unique identification of theevent. A reporting component is an identification of the component thatreported the event. A source component is an identification of thecomponent upon which the event occurred. In many cases, but not all, thereporting component and source component are the same component of thedistributed processing system.

The event and analysis module (124) of FIG. 3 also includes a checkpointmanager (299) that is configured to select during a system shutdownprocedure, a restart incident checkpoint in a distributed processingsystem according to embodiments of the present invention. The checkpointmanager (299) selects during a system shutdown procedure, a restartincident checkpoint according to embodiments of the present invention bydetecting the restart incident checkpoint of each of the incidentanalyzers.

In the example of FIG. 3, the event and alert analysis module (124)includes a monitor (204) that receives events from components of thedistributed processing system and puts the received events (202) in theevent queue (206). The monitor (204) of FIG. 3 may receive events fromcomponents of the distributed processing system on their motion, mayperiodically poll one or more of the components of the distributedprocessing system, or receive events from components in other ways aswill occur to those of skill in the art.

The system of FIG. 3 also includes an event analyzer (208). The eventanalyzer (208) of FIG. 3 is a module of automated computing machineryconfigured to identify alerts in dependence upon received events. Thatis, event analyzers typically receive events and produce alerts. In manyembodiments, a plurality of event analyzers are implemented in parallel.Often event analyzers are assigned to a particular pool of events andmay be focused on events from a particular component or caused by aparticular occurrence to produce a more concise set of alerts.

The event analyzer (208) of FIG. 3 assigns each received event (202) toan event pool (212). An event pool (212) is a collection of eventsorganized by the time of either their occurrence, by the time they arelogged in the event queue, included in the event pool, or other time aswill occur to those of skill in the art. That is, event pools are acollection of events organized by time. Such event pools often providethe ability to analyze a group of time related events to identify alertsin dependence upon them. Often such event pools are useful inidentifying fewer and more relevant alerts in dependence upon multiplerelated events.

In addition, the event analyzer (208) is also configured to selectduring a system shutdown procedure, a restart incident checkpointaccording to embodiments of the present invention. An event analyzerselects during a system shutdown procedure, a restart incidentcheckpoint according to embodiments of the present invention bydetermining whether at least one event is in a queue. If at least oneevent is in the queue, the event analyzer selects as the restart eventcheckpoint, a last event completed checkpoint. If at least one event isnot in the queue, the event analyzer determines whether the last eventcompleted checkpoint matches a last event analysis pool selectioncheckpoint. If the last event completed checkpoint matches a last eventanalysis pool selection checkpoint, the event analyzer selects as therestart event checkpoint, a monitor checkpoint. If the last eventcompleted checkpoint does not match the last event analysis poolselection checkpoint, the event analyzer selects as the restart eventcheckpoint, the last event completed checkpoint.

As mentioned above, in some embodiments of the present invention, morethan one event analyzer may operate in parallel. As such, each eventanalyzer may maintain one or more event pools for selecting during asystem shutdown procedure, a restart incident checkpoint according toembodiments of the present invention. Assigning by the event analyzerthe events to an event pool may therefore include selecting only eventsfrom one or more particular components. In such embodiments, particularcomponents may be selected for a particular event pool to provide eventsassociated with a particular period of time from a particular set of oneor more components.

Assigning by the event analyzer the events to an event pool may also becarried out by selecting only events of a particular event type. In suchembodiments, particular events may be selected for a particular eventpool to provide events associated with a particular period of time froma particular set of event types. Event analyzer (208) in the example ofFIG. 3 identifies in dependence upon the event analysis rules (210) andthe events assigned to the event pool one or more alerts (214). Eventanalyses rules (210) are a collection of predetermined rules formeaningfully parsing received events to identify relevant alerts independence upon the events.

The event analysis rules (210) of FIG. 3 include event arrival rules(230), event pool operation rules (232), event suppression rules (234),and event pool closure rules (236). Event arrival rules (230) areconfigurable predetermined rules for identifying alerts in dependenceupon events in real time as those events are assigned to the event pool.That is, event arrival rules (230) identify alerts in dependence uponevents before closing the event pool. Such rules are typicallypredetermined to identify particular alerts in dependence uponattributes of those events. Event arrival rules may for example dictateidentifying a particular predetermined alert for transmission to asystems administrator in dependence upon a particular event type orcomponent type for the event or other attribute of that event. Suchrules are flexible and may be tailored to a particular distributedcomputing system and its functions.

An alert according to embodiments of the present invention is refinedidentification of an occurrence, such as an error based upon more thanone event, and therefore provides an identification of the occurrence inthe context of its operation in the distributed processing system. Oftenan alert may be a notification of a particular error type of occurrencethat is identified in dependence upon the plurality of events receivedfrom one or more components of the data processing system, such as, forexample, a link failure among a plurality of devices each of which areproducing many events based upon the single link failure, or a powerfailure provoking thousands of events, and so on.

Alerts are often implemented as messages to be sent through a datacommunications network or shared memory. Typical alerts according toembodiments of the present invention have attributes attached to thembased upon the attributes of the events received from which they areidentified.

Event pool operation rules (232) are configurable predetermined rulesfor controlling the operations of the event pool. Such rules includesrules identifying the initial predetermined period of time for eachevent pool, rules dictating the length of time extended to the pool uponthe assignment of each new event to the pool, rules dictating theminimum time an event must be in a pool before that event is included ina collection of events when the pool is closed, rules dictating theamount to extend the initial predetermined period of time based on anarrival rate of events assigned to an event pool, rules governing theclosing of an event pool, and others as will occur to those of skill inthe art. Such rules are flexible and may be tailored to a particulardistributed computing system and its functions.

Event suppression rules (234) are configurable predetermined rules forsuppressing one or more events in a closed pool of events used inidentifying alerts. That is, often events in the closed event pool maybe duplicate events, redundant events, or otherwise unnecessary orunhelpful events in identifying alerts. Such suppression rules aretypically predetermined to delete, drop, or otherwise ignore thosesuppressed events. Event suppression rules may for example dictate thatmore than a threshold number of events of a particular event type orcomponent type are to be suppressed. Such rules are also flexible andmay be tailored to a particular distributed computing system and itsfunctions.

Event pool closure rules (236) are configurable predetermined rules foridentifying alerts in dependence upon unsuppressed events in the closedevent pool and alerts identified by the event arrival rules. That is,event pool closure rules identify new alerts in dependence upon one ormore or even all of the unsuppressed events in the closed event pool.The event pool closure rules also identify alerts in dependence upon thealerts identified by the event arrival rules (230) or a combination ofthe alerts identified by the event arrival rules (230) and one or moreof the unsuppressed events in the closed event pool.

Event analyzer (208) in the example of FIG. 3 sends all the alerts (214)identified by the event analyzer (208) to an alert analyzer (218). Thealert analyzer of FIG. 3 is a module of automated computing machinerycapable of identifying alerts for transmission from events and otheralerts, identifying additional alerts for transmission, and suppressingunnecessary, irrelevant, or otherwise unwanted or unhelpful alertsidentified by the event analyzer. That is, alert analyzers typicallyreceive alerts and events and produce or forward alerts in dependenceupon those alerts and events. In many embodiments, a plurality of alertanalyzers are implemented in parallel. The alerts (216) in the exampleof FIG. 3 are sent from event analyzer (208) to an alert analyzer (218)through an alerts queue (216).

The alert analyzer (218) of FIG. 3 assigns each of the identified alerts(214) to an alert pool (224). An alert pool (224) is a collection ofalerts organized by the time of one or more the events causing the alertto be identified, the time the alert is identified, or other time aswill occur to those of skill in the art. That is, alert pools are acollection of alerts organized by time. Such alert pools often providethe ability to analyze a groups alerts identified and included in thealert pool according to some time. Often such alert pools are useful inidentifying fewer and more relevant alerts in dependence upon multiplerelated events and multiple related alerts.

The alert analyzer (218) of FIG. 3 determines in dependence upon alertanalysis rules (222) and the alerts in the alert pool whether tosuppress any alerts. Suppressing an alert is typically carried out bydropping the alert, deleting the alert or otherwise ignoring or nottransmitting the suppressed alert to a component of the distributedprocessing system.

Alert analyses rules (222) are a collection of rules for suppressing oneor more alerts to provide a more relevant set of alerts for transmissionto a component of the distributed processing system, such as forexample, for display to a systems administrator and to identifyadditional alerts for transmission to one or more components of thedistributed processing system. Alert analysis rules for example maydictate that duplicate alerts are to be suppressed, alerts of aparticular type for transmission to a particular component are to besuppressed, alerts of a particular type be transmitted to a particularcomponent are to be suppressed and so on as will occur to those of skillin the art. Such alerts may be more meaningful to a component of thedistributed processing system for automated error recovery or for asystems administrator who may otherwise be less informed by a number ofraw unanalyzed alerts.

The alert analyzer (218) of FIG. 3 also has access to the events queue(206). The alert analyzer (218) of FIG. 3 in dependence upon the alertanalysis rules may, in some embodiments select events from the eventsqueue and determine whether to suppress any alerts in dependence uponthe selected events. That is, alert analysis rules may also take intoaccount events and their attributes for suppressing alerts and foridentifying additional alerts for transmission to one or morecomponents. Such events may be related to the alerts in the alert poolor independent from such alerts.

The alert analyzer (218) of FIG. 3 transmits the unsuppressed alerts toone or more components of the distributed processing system. The alertanalyzer may transmit the unsuppressed alerts to one or more componentsof the distributed processing system by sending the alert as a messageacross a data communications network, through shared memory, or in otherways as will occur to those of skill in the art. In the example of FIG.3, the unsuppressed alerts (220) are transmitted to a terminal (122) fordisplay to a systems administrator (128).

The alert analyzer (218) of FIG. 3 is also configured to identify independence upon alert analysis rules (222), the alerts in the alert pool(224), and selected events (206) one or more additional alerts andtransmitting the one or more components of the distributed processingsystem. The additional alerts may include one or more alerts notidentified by the event analyzer. Such additional alerts may provideadditional information to a component of the distributed processingsystem of a systems administrator.

In addition, the alert analyzer (218) is also configured to selectduring a system shutdown procedure, a restart incident checkpointaccording to embodiments of the present invention. An alert analyzerselects during a system shutdown procedure, a restart incidentcheckpoint according to embodiments of the present invention bydetermining whether at least one alert is in a queue. If at least onealert is in the queue, the alert analyzer selects as the restart alertcheckpoint, a last alert completed checkpoint. If at least one alert isnot in the queue, the alert analyzer determines whether the last alertcompleted checkpoint matches a last alert analysis pool selectioncheckpoint. If the last alert completed checkpoint matches a last alertanalysis pool selection checkpoint, the alert analyzer selects as therestart alert checkpoint, a monitor checkpoint. If the last alertcompleted checkpoint does not match the last alert analysis poolselection checkpoint, the alert analyzer selects as the restart alertcheckpoint, the last alert completed checkpoint.

As mentioned above, selecting during a system shutdown procedure, arestart incident checkpoint according to embodiments of the presentinvention may include assigning events to an event pool and those poolsare administered according to embodiments of the present invention. Forfurther explanation, FIG. 4 sets forth a diagram illustrating assigningevents to an event pool according to embodiments of the presentinvention. An event pool (212) is a collection of events organized bythe time of either their occurrence, by the time they are logged in theevent queue, included in the event pool, or other time as will occur tothose of skill in the art. That is, event pools are a collection ofevents organized by time. Such event pools often provide the ability toanalyze a group of time related events and to identify alerts independence upon them. Often such event pools are useful in identifyingfewer and more relevant alerts in dependence upon multiple relatedevents.

Event pools according to embodiments of the present invention aretypically operated according to event pool operation rules which arethemselves often included in event analysis rules. Such event pooloperation rules are configurable predetermined rules for controlling theoperations of the event pool. Such rules includes rules identifying theinitial predetermined period of time for each event pool, rulesdictating the length of time extended to the pool upon the assignment ofeach new event to the pool, rules dictating the minimum time an eventmust be in a pool before that event is included in a collection ofevents when the pool is closed, rules dictating the amount to extend theinitial predetermined period of time based on an arrival rate of eventsassigned to an event pool, rules governing the closing of an event pool,and others as will occur to those of skill in the art. Such rules areflexible and may be tailored to a particular distributed computingsystem and its functions.

Events are often assigned to an event pool according to their loggedtime. That is, events are typically inserted into the event pool in theorder that they are received in the event queue. In the example of FIG.4, the timing of the event pool (212) is initiated when the first event‘Event 0’ (400) is assigned to the event pool (212) at time t₀. Theevent pool of FIG. 4 is initiated for a predetermined initial period oftime from t₁ to t_(f). That is, upon receiving the first event ‘Event 0’(400) the event pool of FIG. 4 has a predetermined initial period oftime beginning at t₁ and ending at t_(f). The predetermined initialperiod of time may be configured in dependence upon a number of factorsas will occur to those of skill in the art such as, the number ofcomponents in the distributed processing system, the frequency ofreceiving events, the types of events typically received and so on aswill occur to those of skill in the art.

In the example FIG. 4, the initial period of time is extended for eachnew event assigned to the event pool during the predetermined initialperiod from t₁ to t_(f) by a particular period of time assigned to theevent. In the example of FIG. 4 upon assigning ‘Event l’ (402) to theevent pool (212) the predetermined initial period of time t₀-t_(f) isextended by ‘Extension 1’ (406) having a time of e1 thereby creating anew time for closing the event pool (212) at t_(f+e1) if no other eventsare assigned to the pool before t_(f+e1). Similarly, in the example ofFIG. 4 upon assigning ‘Event 2’ (404) to the event pool having a time ofe2, the now extended period of time from t₀-t_(f+e1) is extended againby extension 2 (406) thereby establishing a new time for closing thepool at time t_(f−e1−e2) if no other events are assigned to the poolbefore t_(f+e1+e2) or before some maximum time for the event pool hasexpired. In this manner, the event pool is extended with each receivedevent until a collection of events that may be usefully used to identifyalerts is assigned to the event pool. According to embodiments of thepresent invention, the predetermined initial period of time may beextended based on an arrival rate at which events are assigned to anevent pool.

In typical embodiments of the present invention, event pools may have amaximum duration that can no longer be extended. In such cases, arequirement may exist that an event that has not resided in the eventpool for a threshold period of time be moved to a next event pool. Insome embodiments, the attributes of such an event that is moved to thenext event pool are used for relevant alert delivery with the initialevent pool and in other embodiments; the attributes of such an event areused for relevant alert delivery with the next event pool to which thatevent is moved.

In the example of FIG. 4, when conditions are met to close the pool anevents analyzer determines for each event (400, 402, 404) in the pool(212) whether the event has been in the pool for its predeterminedminimum time for inclusion in a pool. If the event has been in the poolfor its predetermined minimum time, the event is included in the closedpool for event analysis for relevant alert delivery. If the event hasnot been in the pool for its predetermined minimum time, the event isevicted from the closed pool and included a next pool for event analysisfor relevant alert delivery.

In many embodiments, a plurality of event pools may be used in paralleland one or more of such event pools are assigned to a particular eventsanalyzer. In such embodiments, events analyzers may be directed toevents in event pools having particular attributes.

As mentioned above, selecting during a system shutdown procedure, arestart incident checkpoint according to embodiments of the presentinvention may include assigning alerts to an alert pool and those poolsare administered according to embodiments of the present invention. Forfurther explanation, FIG. 5 sets forth a diagram illustrating assigningalerts to an alert pool according to embodiments of the presentinvention. The alert pool (224) of FIG. 5 operates in a manner similarto the event pool of FIG. 4. That is, the alert pool according to theexample of FIG. 5 includes alerts and the timing of the alert poolbegins with the first alert ‘Alert 0’ (500) at time t₀ and is configuredto have a predetermined initial period of time t₀-tf. In the example ofFIG. 5, the initial period of time is extended for each new alertassigned to the alert pool in the predetermined initial period from t₁to t_(f) by a particular period of time assigned to the alert. In theexample of FIG. 5, upon assigning ‘Alert l’ (502) to the alert pool(224) the predetermined initial period of time t₀-t_(f) is extended by‘Extension 1’ (506) having a time of e1 thereby creating a new time forclosing the alert pool (224) at t_(f+e1) if no other alerts are assignedto the pool before t_(f+e1). Similarly, in the example of FIG. 4 uponassigning ‘Alert 2’ (504) to the alert pool having a time of e2, the nowextended period of time from t₀-t_(f+e1) is extended again by ‘Extension2’ (406) thereby establishing a new time for closing the pool at timet_(f+e1+e2) if no other alerts are assigned to the pool beforet_(f+e1+e2) or before some maximum time for the alert pool has expired.According to embodiments of the present invention, the predeterminedinitial period of time may be extended based on an arrival rate at whichalerts are assigned to an alert pool.

In typical embodiments of the present invention, alert pools may have amaximum duration that can no longer be extended. In such cases, arequirement may exist that an alert that has not resided in the alertpool for a threshold period of time be moved to a next alert pool. Insome embodiments, the attributes of such an alert that is moved to thenext alert pool are used for relevant alert delivery according toembodiments of the present invention with the initial alert pool and inother embodiments, the attributes of such an alert are used for relevantalert delivery with the next alert pool to which that alert is moved.

In the example of FIG. 5, when conditions are met to close the pool analerts analyzer determines for each alert (500, 502, 504) in the pool(224) whether the alert has been in the pool for its predeterminedminimum time for inclusion in a pool. If the alert has been in the poolfor its predetermined minimum time, the alert is included in the closedpool for alert analysis for relevant alert delivery according toembodiments of the present invention. If the alert has not been in thepool for its predetermined minimum time, the alert is evicted from theclosed pool and included a next pool for alert analysis for relevantalert delivery according to embodiments of the present invention.

In many embodiments, a plurality of alert pools may be used in paralleland one or more of such alert pools are assigned to a particular alertsanalyzer. In such embodiments, alerts analyzers may be directed toalerts in alert pools having particular attributes.

As mentioned above, selecting during a system shutdown procedure, arestart incident checkpoint according to embodiments of the presentinvention may include the administration of one or more pools ofincidents such as events, alerts or other incidents as will occur tothose of skill in the art. For further explanation, FIG. 6 sets forth aflow chart illustrating an example method of selecting during a systemshutdown procedure, a restart incident checkpoint for incident analysisin a distributed processing system in a distributed processing systemaccording to embodiments of the present invention. The method of FIG. 6includes receiving (402) in an event queue a plurality of events (202)from one or more components of a distributed processing system.Attributes of events useful in selecting during a system shutdownprocedure, a restart incident checkpoint for incident analysis in adistributed processing system according to embodiments of the presentinvention may include an occurred time, a logged time, an event type, anevent ID, a reporting component, and a source component.

Receiving (402) in an event queue a plurality of events (202) from oneor more components of a distributed processing system may be carried outby receiving an event initiated by one or more components of the dataprocessing system and storing the event in the event queue according tothe time in which the event occurred or according to the time the eventwas received. Receiving (402) in an event queue a plurality of events(202) from one or more components of a distributed processing systemalso may be carried out by polling a component for status and receivingin response an event and storing the event in the event queue accordingto the time in which the event occurred or according to the time theevent was received.

The method of FIG. 6 also includes assigning (404) by an event analyzereach received event to an event pool (212). In some embodiments of thepresent invention, assigning (404) by an event analyzer each receivedevent (202) to an event pool (212) may be carried out by assigningevents to the event pool according to the logged time. Assigning (404)by an event analyzer each received event (202) to an event pool (212)may also be carried out in dependence upon attributes of the event. Suchattributes may include an identification or type of the component uponwhich an occurrence occurred to create the event, the reportingcomponent of the event, the event ID, the event type, and so on as willoccur to those of skill in the art.

An event pool according to the method of FIG. 6 includes eventsoccurring during a predetermined initial period of time and in theexample of FIG. 6 assigning (404) by the event analyzer each receivedevent to an event pool includes extending (426) for each event assignedto the event pool the predetermined initial period of time by aparticular period of time assigned to the event.

The event analyzer includes event analysis rules (210) including, eventarrival rules, event pool operation rules, event suppression rules, andevent pool closure rules. Event arrival rules are configurablepredetermined rules for identifying alerts in dependence upon events inreal time as those events are assigned to the event pool. That is, eventarrival rules identify alerts in dependence upon events before closingthe event pool. Such rules are flexible and may be tailored to aparticular distributed computing system and its functions.

An alert according to embodiments of the present invention is refinedidentification of an occurrence, such and an error based upon more thanone event and therefore provides an identification of the occurrence inthe context of its operation in the distributed processing system. Oftenan alert may be a notification of a particular error type of occurrencethat is identified in dependence upon the plurality of events receivedfrom one or more components of the data processing system, such as, forexample, a link failure among a plurality of devices each of which areproducing many events based upon the single link failure, or a powerfailure provoking thousands of events, and so on.

Alerts are often implemented as messages to be sent through a datacommunications network or shared memory. Typical alerts according toembodiments of the present invention have attributes attached to thembased upon the attributes of the events received from which they areidentified.

Event pool operation rules are configurable predetermined rules forcontrolling the operations of the event pool. Such rules includes rulesidentifying the initial predetermined period of time for each eventpool, rules dictating the length of time extended to the pool upon theassignment of each new event to the pool, rules dictating the minimumtime an event must be in a pool before that event is included in acollection of events when the pool is closed, rules governing theclosing of an event pool, and others as will occur to those of skill inthe art. Such rules are flexible and may be tailored to a particulardistributed computing system and its functions.

Event suppression rules are configurable predetermined rules forsuppressing one or more events in a closed pool of events used inidentifying alerts. That is, often events in the closed event pool maybe duplicate events, redundant events, or otherwise unnecessary orunhelpful events in identifying alerts. Such suppression rules aretypically predetermined to delete, drop, or otherwise ignore thosesuppressed events. Event suppression rules may for example dictate thatmore than a threshold number of events of a particular event type orcomponent type are to be suppressed. Such rules are also flexible andmay be tailored to a particular distributed computing system and itsfunctions.

Event pool closure rules are configurable predetermined rules foridentifying alerts in dependence upon unsuppressed events in the closedevent pool and alerts identified by the event arrival rules. That is,event pool closure rules identify new alerts in dependence upon one ormore or even all of the unsuppressed events in the closed event pool.The event pool closure rules also identify alerts in dependence upon thealerts identified by the event arrival rules or a combination of thealerts identified by the event arrival rules and one or more of theunsuppressed events in the closed event pool.

The method of FIG. 6 also includes identifying (410) by the eventanalyzer in dependence upon the event arrival rules and the eventsassigned to the event pool one or more alerts (214). Identifying (410)by the event analyzer in dependence upon the event arrival rules and theevents assigned to the event pool one or more alerts (214) may becarried out by identifying alerts in dependence upon one or moreattributes of the events as that event is assigned to the event pool.Identifying (410) by the event analyzer in dependence upon the eventarrival rules and the events assigned to the event pool one or morealerts (214) may be carried by comparing the attributes of the events tothe event arrival rules and identifying as a result of the comparisonone or more alerts. Such attributes may include the type of componentfrom which the event was received, the type of component creating theevent, the identification of the component creating the event, the timethe event was created or received, an error reported in the event, andmany others as will occur to those of skill in the art.

The method of FIG. 6 also includes closing (412), by the event analyzerin dependence upon the event pool operation rules, the event pool (212).Closing (412), by the event analyzer in dependence upon the event pooloperation rules, the event pool (212) may be carried out by determiningthat conditions dictated by the event pool operation rules have been metto stop assigning new events to the event pool and identifying independence upon those event pool operation rules the particular eventsthat are included in the closed pool of events.

Closing the event pool may be carried out by determining that theinitial period of time for the event pool and any particular periods oftime for events received in the event pool extended to the initialperiod of time have expired. In such cases, if no new events arereceived prior to the expiration of the initial period of time for theevent pool and any particular periods of time for events received in theevent pool extended to the initial period of time the pool is closed.

Closing the event pool may also be carried out by determining that amaximum duration for the event pool has expired. In such cases,regardless of the number of new events being received after a maximumduration for the event pool has expired, the pool is closed. In suchembodiments, a maximum duration for the event pool prevents the eventpool from including more events than are useful for relevant alertdelivery according to embodiments of the present invention.

The method of FIG. 6 also includes determining (414), by the eventsanalyzer in dependence upon the event suppression rules, whether tosuppress one or more events in the closed event pool (212). Determining(414), by the events analyzer in dependence upon the event suppressionrules, whether to suppress one or more events in the closed event pool(212) may be carried out by determining in dependence upon theattributes of one or more events in the closed pool whether to delete,drop, or otherwise ignore one or more of the events in the closed pool.

The method of FIG. 6 includes identifying (416) by the event analyzer independence upon the event pool closure rules and any unsuppressed eventsassigned to the event pool, one or more additional alerts (417).Identifying (416) by the event analyzer in dependence upon the eventpool closure rules and any unsuppressed events assigned to the eventpool, one or more additional alerts (417) may be carried out byidentifying alerts in dependence upon one or more attributes of theevents as that event is assigned to the event pool. Identifying (416) bythe event analyzer in dependence upon the event pool closure rules andany unsuppressed events assigned to the event pool, one or moreadditional alerts (417) may be carried out by selecting the unsuppressedevents for the event pool, comparing the attributes of the unsuppressedevents of the event pool to the pool closure rules, and identifying as aresult of the comparison one or more additional alerts. Such attributesmay include the type of component from which one or more of theunsuppressed events are received, the type of components creating theunsuppressed events, the identification of the component creating theunsuppressed events, the time the events were created or received, oneor more errors reported by the events event, the number of events in thepool, and many others as will occur to those of skill in the art.

The method of FIG. 6 includes sending (418) by the event analyzer to analert analyzer all the alerts identified by the event analyzer. Sending(418) by the event analyzer to an alert analyzer all the alerts (214)identified by the event analyzer may be carried out by sending a messagecontaining the alerts from the event analyzer to the alert analyzer.Such a message may be sent from the event analyzer to the alert analyzeracross a network, through shared memory, or in other ways as will occurto those of skill in the art.

The method of FIG. 6 includes assigning (420) by the alert analyzer theidentified alerts to an alert pool (224). An alert pool according to themethod of FIG. 6 has a predetermined initial period of time and in theexample of FIG. 6 assigning (420) by the alert analyzer the identifiedalerts to an alert pool (224) includes extending for each alert assignedto the alert pool the predetermined initial period of time by aparticular period of time assigned to the alert. Assigning (420) by thealert analyzer the identified alerts to an alert pool (224) also may becarried out in dependence upon attributes of the alerts. Such attributesmay include an identification or type of the component upon which anoccurrence occurred to create the event that was used to identify thealert, the alert ID, the alert type, and so on as will occur to those ofskill in the art.

The method of FIG. 6 includes determining (422) by the alert analyzer independence upon alert analysis rules (222) and the alerts in the alertpool whether to suppress any alerts. Determining (422) by the alertanalyzer in dependence upon alert analysis rules (222) and the alerts inthe alert pool whether to suppress any alerts may be carried out independence upon one or more attributes of the alerts. Such attributesmay include an identification or type of the component upon which anoccurrence occurred to create the event that was used to identify thealert, the alert ID, the alert type, and so on as will occur to those ofskill in the art. In such embodiments, determining (422) by the alertanalyzer in dependence upon alert analysis rules (222) and the alerts inthe alert pool whether to suppress any alerts may be carried out bycomparing the attributes of the alerts in the alert pool to the alertanalysis rules and identifying as a result of the comparison one or morealerts for suppression according to the event analysis rules.

The method of FIG. 6 includes transmitting (420) the unsuppressed alertsto one or more components of the distributed processing system.Transmitting (420) the unsuppressed alerts to one or more components ofthe distributed processing system may be carried out by sending amessage containing the alert to one or more components of thedistributed processing system. In many cases, an alert may be sent as amessage to a systems administrator advising the systems administrator ofone or more occurrences within the distributed processing system.

As mentioned above, alert analysis rules may select additional alerts orsuppress alerts in dependence upon events. In such embodiments,determining whether to suppress any alerts includes selecting events anddetermining whether to suppress any alerts in dependence upon theselected events. The method of FIG. 6 therefore also includesidentifying (426) by the alert analyzer in dependence upon alertanalysis rules (222), the alerts in the alert pool (224), and anyselected events one or more additional alerts and in the method of FIG.6, transmitting (428) the unsuppressed alerts also includes transmitting(430) any additional alerts to one or more components of the distributedprocessing system.

For further explanation, FIG. 7 sets forth a flow chart illustrating anexemplary method of selecting during a system shutdown procedure, arestart incident checkpoint of an incident analyzer in a distributedprocessing system according to embodiments of the present invention. Asystem shutdown procedure may include shutting down one or morecomponents of a distributed processing system. For example, an incidentanalyzer may receive an instruction to shut down immediately and inresponse to which, the incident analyzer selects a restart incidentcheckpoint according to the method of FIG. 7. A restart incidentcheckpoint identifies an incident for the incident analyzer to use uponrestart of the incident analyzer after completion of the system shutdownprocedure. As explained above, use of the term ‘incident’ may refer toeither an event or alert. That is, an incident analyzer may be an eventanalyzer or an alert analyzer.

The method of FIG. 7 includes an incident analyzer (700) determining(702) during a system shutdown procedure, whether at least one incidentis in the incident receive queue (770). In the example of FIG. 7, theincident receive queue (770) is configured to provide incidents to anincident analysis pool (772). In a particular embodiment, the incidentreceive queue (770) is provided incidents from an incident queue of thedistributed processing system. Determining (702) whether at least oneincident is in the incident receive queue (770) may be carried out byexamining the contents of the incident receive queue (770).

If at least one incident is in the incident receive queue (770), themethod of FIG. 7 continues by the incident analyzer (700) selecting(704) as the restart incident checkpoint (750), a last incidentcompleted checkpoint (751). The incident analyzer may be configured tomaintain a plurality of checkpoints that indicate processing ofincidents by the incident analyzer. Processing of incidents may includemoving incidents from an incident receive queue to an incident analysispool, closing the incident analysis pool, generating one or more eventsor alerts based on incidents within the incident analysis pool, andremoving the incidents from the incident analysis pool upon completionof processing of the incidents within the incident analysis pool. A lastincident completed checkpoint identifies the last incident included inan incident analysis pool that has completed processing. That is, afterthe incident analyzer has completed processing of the incidents withinthe incident analysis pool, the incident analyzer updates the lastincident completed checkpoint to indicate an identification of the lastincident that was included in the incident analysis pool.

If at least one incident is not in the incident receive queue (770), themethod of FIG. 7 continues by the incident analyzer (700) determining(706) whether the last incident completed checkpoint (752) matches alast incident analysis pool selection checkpoint (751). In addition tothe last incident completed checkpoint (751), the incident analyzer(700) may also be configured to maintain a last incident analysis poolselection checkpoint. A last incident analysis pool selection checkpointidentifies the last incident selected for inclusion in the incidentanalysis pool. That is, as incidents are added to an incident pool, theincident analyzer updates the last incident analysis pool selectioncheckpoint to indicate the last incident added to the pool regardless ofwhether processing of the incident analysis pool has completedprocessing. Determining (706) whether the last incident completedcheckpoint (752) matches the last incident analysis pool selectioncheckpoint (751) may be carried out by comparing the identifications ofincidents indicated by the last incident completed checkpoint and thelast incident analysis pool selection checkpoint.

If the last incident completed checkpoint (752) matches a last incidentanalysis pool selection checkpoint (751), the method of FIG. 7 continuesby the incident analyzer selecting (708) as the restart incidentcheckpoint (750), a monitor checkpoint (754). A distributed processingsystem may include a plurality of incident analyzers. The incidentanalyzers may be each provided all of the incidents, however, not all ofthe incident analyzers may be configured to process all of theincidents. That is, some incident analyzer may be configured to onlyplace incidents of a certain type within an incident receive queue andan incident analysis pool and to ignore other types of incidents.

For example, an incident queue of the distributed processing system mayprovide to both a first incident analyzer and a second incidentanalyzer, a first set of incidents followed by a subsequent second setof incidents. A monitor may be configured to maintain a monitorcheckpoint indicating the last incident that each of the incidentanalyzers received. In this example, the first incident analyzer and thesecond incident analyzer are both configured to process the first set ofincidents but only the second incident analyzer is configured to processthe second subsequent set of incidents. Continuing with this example,when both incident analyzers have completed processing of theirincidents, the first incident analyzer may indicate checkpoints (e.g.,the last incident analysis pool selection checkpoint and the lastincident completed checkpoint) corresponding to completion of up to onlythe first set of incidents and the second incident analyzer may indicatecheckpoints corresponding to completion up to the second set ofincidents. In this example, according to embodiments of the presentinvention, the first incident analyzer may select the monitor checkpointas the restart incident checkpoint, which indicates the last incident ofthe second set of incident. Selecting the monitor checkpoint as therestart incident checkpoint prevents the incident analyzer from havingto re-receive incidents received between the last incident analysis poolselection checkpoint and the monitor checkpoint.

If the last incident completed checkpoint (752) does not match the lastincident analysis pool selection checkpoint (751), the method of FIG. 7continues by the incident analyzer (700) selecting (710) as the restartincident checkpoint (750), the last incident completed checkpoint (752).When the last incident completed checkpoint does not match the lastincident analysis pool selection checkpoint, the incident analyzer isstill processing incidents within the incident analysis pool. If theincident analyzer is still processing incidents within the incidentanalysis pool during a shutdown procedure, processing of the incidentwill not be completed before shutdown. Accordingly, the last incidentanalysis pool selection checkpoint is selected as the restart incidentcheckpoint. That is, after restart, the incident analyzer willre-receive with the incident receive queue, the incidents received afterthe last incident analysis pool selection checkpoint and move theincidents into the incident analysis pool.

For further explanation, FIG. 8 sets forth a flow chart illustrating anadditional method of selecting during a system shutdown procedure, arestart incident checkpoint of an incident analyzer in a distributedprocessing system according to embodiments of the present invention. Themethod of FIG. 8 is similar to the method of FIG. 7 in that the methodof FIG. 8 includes: determining (702) during a system shutdownprocedure, whether at least one incident is in the incident receivequeue (770); if at least one incident is in the incident receive queue(770), selecting (704) as the restart incident checkpoint (750), a lastincident completed checkpoint (751); and if at least one incident is notin the incident receive queue (770): determining (706) whether the lastincident completed checkpoint (752) matches a last incident analysispool selection checkpoint (751); if the last incident completedcheckpoint (752) matches a last incident analysis pool selectioncheckpoint (751), selecting (708) as the restart incident checkpoint(750), a monitor checkpoint (754); and if the last incident completedcheckpoint (752) does not match the last incident analysis poolselection checkpoint (751), selecting (710) as the restart incidentcheckpoint (750), the last incident completed checkpoint (752).

The method of FIG. 8 also includes the incident analyzer (700) recording(802) as the last incident analysis pool selection checkpoint (751), anidentification (850) of a last incident selected for inclusion in theincident analysis pool (772). Recording (802) as the last incidentanalysis pool selection checkpoint (751), an identification (850) of alast incident selected for inclusion in the incident analysis pool (772)may be carried out by updating the last incident analysis pool selectioncheckpoint with every addition of incident to the incident analysispool.

The method of FIG. 8 includes the incident analyzer (700), uponcompletion of processing of incidents within the incident analysis pool(772), recording (804) as the last incident completed checkpoint (752),an identification (852) of a last incident included in the incidentanalysis pool (772). Recording (804) as the last incident completedcheckpoint (752), an identification (852) of a last incident included inthe incident analysis pool (772) may be carried out by determining thatanalysis of incidents within an incident analysis pool is complete; andin response to determining that analysis is complete, updating the lastincident completed checkpoint with the last incident included in theincidents that just completed processing within the incident analysispool.

For further explanation, FIG. 9 sets forth a flow chart illustrating anadditional method of selecting during a system shutdown procedure, arestart incident checkpoint of an incident analyzer in a distributedprocessing system according to embodiments of the present invention. Themethod of FIG. 9 is similar to the method of FIG. 7 in that the methodof FIG. 9 includes: determining (702) during a system shutdownprocedure, whether at least one incident is in the incident receivequeue (770); if at least one incident is in the incident receive queue(770), selecting (704) as the restart incident checkpoint (750), a lastincident completed checkpoint (751); and if at least one incident is notin the incident receive queue (770): determining (706) whether the lastincident completed checkpoint (752) matches a last incident analysispool selection checkpoint (751); if the last incident completedcheckpoint (752) matches a last incident analysis pool selectioncheckpoint (751), selecting (708) as the restart incident checkpoint(750), a monitor checkpoint (754); and if the last incident completedcheckpoint (752) does not match the last incident analysis poolselection checkpoint (751), selecting (710) as the restart incidentcheckpoint (750), the last incident completed checkpoint (752).

The method of FIG. 9 also includes a monitor (900) selecting (902), asthe monitor checkpoint (754), an identification (980) of the lastincident provided to all incident analyzer receive queues (976, 770) ofall the incident analyzers (974, 700). In the example of FIG. 9, themonitor (900) is configured to receive incidents from the distributedprocessing system and provide the incidents to a plurality of incidentanalyzer receive queues (976, 770) of a plurality of incident analyzers(700, 974). For example, a monitor may send to the incident analyzers, astream of incidents beginning with the incident identified by therestart incident checkpoint and continuing with any incidents having anewer identification number than the incident identified by the restartincident checkpoint. The monitor may send the stream of incidents byinstructing the incident database (970) to send the identified incidentsto the incident queue (972), which then sends the incidents to theplurality of incident receive queues (976, 770). Selecting (902), as themonitor checkpoint (754), an identification (980) of the last incidentprovided to all incident analyzer receive queues (976, 770) of all theincident analyzers (974, 700) may be carried out by determining when anincident has been provided to each of the incident analyzers; and inresponse to determining that the incident has been provided to each ofthe incident analyzers, storing an identification of that incident asthe monitor checkpoint.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

It will be understood from the foregoing description that modificationsand changes may be made in various embodiments of the present inventionwithout departing from its true spirit. The descriptions in thisspecification are for purposes of illustration only and are not to beconstrued in a limiting sense. The scope of the present invention islimited only by the language of the following claims.

What is claimed is:
 1. A method of selecting during a system shutdownprocedure, a restart incident checkpoint of an incident analyzer in adistributed processing system, the restart incident checkpointidentifying an incident for the incident analyzer to use upon restart ofthe incident analyzer, the incident analyzer including an incidentanalysis pool and an incident receive queue to provide incidents to theincident analysis pool, the method comprising: determining during thesystem shutdown procedure, by the incident analyzer, whether at leastone incident is in the incident receive queue; if at least one incidentis in the incident receive queue, selecting as the restart incidentcheckpoint, a last incident completed checkpoint; and if at least oneincident is not in the incident receive queue: determining whether thelast incident completed checkpoint matches a last incident analysis poolselection checkpoint; if the last incident completed checkpoint matchesthe last incident analysis pool selection checkpoint, selecting as therestart incident checkpoint, a monitor checkpoint; and if the lastincident completed checkpoint does not match the last incident analysispool selection checkpoint, selecting as the restart incident checkpoint,the last incident completed checkpoint.
 2. The method of claim 1 furthercomprising recording, by the incident analyzer, as the last incidentanalysis pool selection checkpoint, an identification of a last incidentselected for inclusion in the incident analysis pool.
 3. The method ofclaim 1 further comprising upon completion of processing of incidentswithin the incident analysis pool, recording, by the incident analyzer,as the last incident completed checkpoint, an identification of a lastincident included in the incident analysis pool.
 4. The method of claim1 wherein a monitor is configured to receive incidents from thedistributed processing system and provide the incidents to a pluralityof incident analyzer receive queues of a plurality of incidentanalyzers; and wherein the method further comprises selecting, by themonitor, as the monitor checkpoint, an identification of a last incidentprovided to all incident analyzer receive queues of all the incidentanalyzers.
 5. The method of claim 1 wherein the incident analyzer is anevent analyzer, the incident receive queue is an event receive queue,and the incident analysis pool is an event analysis pool.
 6. The methodof claim 1 wherein the incident analyzer is an alert analyzer, theincident receive queue is an alert receive queue, and the incidentanalysis pool is an alert analysis pool.
 7. An apparatus for selectingduring a system shutdown procedure, a restart incident checkpoint of anincident analyzer in a distributed processing system, the restartincident checkpoint identifying an incident for the incident analyzer touse upon restart of the incident analyzer, the incident analyzerincluding an incident analysis pool and an incident receive queue toprovide incidents to the incident analysis pool, the apparatuscomprising a computer processor and a computer memory operativelycoupled to the computer processor, the computer memory having disposedwithin it computer program instructions that when executed by thecomputer processor cause the apparatus to carry out the steps of:determining during the system shutdown procedure, by the incidentanalyzer, whether at least one incident is in the incident receivequeue; if at least one incident is in the incident receive queue,selecting as the restart incident checkpoint, a last incident completedcheckpoint; and if at least one incident is not in the incident receivequeue: determining whether the last incident completed checkpointmatches a last incident analysis pool selection checkpoint; if the lastincident completed checkpoint matches the last incident analysis poolselection checkpoint, selecting as the restart incident checkpoint, amonitor checkpoint; and if the last incident completed checkpoint doesnot match the last incident analysis pool selection checkpoint,selecting as the restart incident checkpoint, the last incidentcompleted checkpoint.
 8. The apparatus of claim 7 further comprisingcomputer program instructions that when executed by the computerprocessor cause the apparatus to carry out the steps of recording, bythe incident analyzer, as the last incident analysis pool selectioncheckpoint, an identification of a last incident selected for inclusionin the incident analysis pool.
 9. The apparatus of claim 7 furthercomprising computer program instructions that when executed by thecomputer processor cause the apparatus to carry out the steps of: uponcompletion of processing of incidents within the incident analysis pool,recording, by the incident analyzer, as the last incident completedcheckpoint, an identification of a last incident included in theincident analysis pool.
 10. The apparatus of claim 7 further comprising:a monitor configured to receive incidents from the distributedprocessing system and provide the incidents to a plurality of incidentanalyzer receive queues of a plurality of incident analyzers; andcomputer program instructions that when executed by the computerprocessor cause the apparatus to carry out the steps of selecting, bythe monitor, as the monitor checkpoint, an identification of a lastincident provided to all incident analyzer receive queues of all theincident analyzers.
 11. The apparatus of claim 7 wherein the incidentanalyzer is an event analyzer, the incident receive queue is an eventreceive queue, and the incident analysis pool is an event analysis pool.12. The apparatus of claim 7 wherein the incident analyzer is an alertanalyzer, the incident receive queue is an alert receive queue, and theincident analysis pool is an alert analysis pool.
 13. A computer programproduct for selecting during a system shutdown procedure, a restartincident checkpoint of an incident analyzer in a distributed processingsystem, the restart incident checkpoint identifying an incident for theincident analyzer to use upon restart of the incident analyzer, theincident analyzer including an incident analysis pool and an incidentreceive queue to provide incidents to the incident analysis pool, thecomputer program product disposed upon a computer readable storagemedium, the computer program product comprising computer programinstructions that when executed by a computer cause the computer tocarry out the steps of: determining during the system shutdownprocedure, by the incident analyzer, whether at least one incident is inthe incident receive queue; if at least one incident is in the incidentreceive queue, selecting as the restart incident checkpoint, a lastincident completed checkpoint; and if at least one incident is not inthe incident receive queue: determining whether the last incidentcompleted checkpoint matches a last incident analysis pool selectioncheckpoint; if the last incident completed checkpoint matches the lastincident analysis pool selection checkpoint, selecting as the restartincident checkpoint, a monitor checkpoint; and if the last incidentcompleted checkpoint does not match the last incident analysis poolselection checkpoint, selecting as the restart incident checkpoint, thelast incident completed checkpoint.
 14. The computer program product ofclaim 13 further comprising computer program instructions that whenexecuted by a computer cause the computer to carry out the steps of:recording, by the incident analyzer, as the last incident analysis poolselection checkpoint, an identification of a last incident selected forinclusion in the incident analysis pool.
 15. The computer programproduct of claim 13 further comprising computer program instructionsthat when executed by a computer cause the computer to carry out thesteps of: upon completion of processing of incidents within the incidentanalysis pool, recording, by the incident analyzer, as the last incidentcompleted checkpoint, an identification of a last incident included inthe incident analysis pool.
 16. The computer program product of claim 13wherein the distributed processing system includes a monitor configuredto receive incidents from the distributed processing system and providethe incidents to a plurality of incident analyzer receive queues of aplurality of incident analyzers; and wherein the computer programproduct further comprises computer program instructions that whenexecuted by a computer cause the computer to carry out the steps of:selecting, by the monitor, as the monitor checkpoint, an identificationof a last incident provided to all incident analyzer receive queues ofall the incident analyzers.
 17. The computer program product of claim 13wherein the incident analyzer is an event analyzer, the incident receivequeue is an event receive queue, and the incident analysis pool is anevent analysis pool.
 18. The computer program product of claim 13wherein the incident analyzer is an alert analyzer, the incident receivequeue is an alert receive queue, and the incident analysis pool is analert analysis pool.
 19. The computer program product of claim 13,wherein the computer readable medium further comprises a computerreadable signal medium.
 20. The computer program product of claim 13,wherein the computer readable medium further comprises a computerreadable storage medium.